Terminal device, virtual object manipulation method, and virtual object manipulation program

ABSTRACT

A terminal device according to one aspect presents an augmented reality space to a user. The terminal device includes: an image-capturing unit configured to capture an image of a real space; a display unit configured to display an augmented reality space image representing the augmented reality space including the real space captured by the image-capturing unit and a virtual object; a determination unit configured to determine at least a part of the virtual object as an operation target; and an object control unit configured to control an operation of the operation target in the augmented reality space. The object control unit detects a direction and an amount of a movement of the terminal device after the operation target is determined and moves the operation target based on the detected direction and amount of the movement of the terminal device.

TECHNICAL FIELD

An aspect of the present disclosure relates to a terminal device, avirtual object operation method, and a virtual object operation program.

BACKGROUND ART

There is known a mechanism of providing users with an augmented realityspace which is obtained by superimposing virtual objects on a realspace. For example, Patent Document 1 discloses a mechanism ofdisplaying indicators for adjusting parameters for components of avirtual object (i.e., a 3D object) on a screen, receiving an instructionfrom a user via an indicator, and adjusting, according to theinstruction, the position of the component corresponding to theindicator. Patent Document 1 illustrates, as a specific example, amechanism of vertically adjusting the position of the nose of a virtualobject upward or downward by moving an indicator for the nose of thevirtual object rightward or leftward.

CITATION LIST Patent Document

PATENT DOCUMENT 1: Japanese Unexamined Patent Publication No. 2020-46863

SUMMARY OF THE INVENTION Technical Problems

The mechanism disclosed in Patent Document 1 adjusts the position of acomponent of a virtual object through a user operation via an indicator.In the mechanism using the indicator displayed on a screen, however,only two-dimensional operations (e.g., an operation of sliding theindicator rightward or leftward as described above) are possible, thatis, intuitive position adjustment of the component in thethree-dimensional space is impossible.

To address the problem, it is an object of one aspect of the presentdisclosure to provide a terminal device, a virtual object operationmethod, and a virtual object operation program capable of allowingintuitive position adjustment of a virtual obj ect in an augmentedreality space.

Solution to the Problems

A terminal device according to one aspect of the present disclosure is aterminal device configured to present an augmented reality space to auser, the terminal device including: an image-capturing unit configuredto capture an image of a real space; a display unit configured todisplay an augmented reality space image representing the augmentedreality space including the real space captured by the image-capturingunit and a virtual object; a determination unit configured to determineat least a part of the virtual object as an operation target; and anobject control unit configured to control an operation of the operationtarget in the augmented reality space, wherein the object control unitis configured to: detect a direction and an amount of a movement of theterminal device after the operation target is determined; and move theoperation target based on the detected direction and amount of themovement of the terminal device.

Advantages of the Invention

An aspect of the present disclosure allows intuitive position adjustmentof a virtual object in an augmented reality space.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example hardware configuration of a user terminalthat is a terminal device according to an embodiment.

FIG. 2 illustrates an example functional configuration of the userterminal illustrated in FIG. 1 .

FIG. 3 is a flowchart illustrating an example processing procedure of amoving operation of a character object using the user terminal in FIG. 1.

FIG. 4 illustrates an example moving operation of the character object.

FIG. 5 illustrates the example moving operation of the character object.

FIG. 6 is a flowchart illustrating an example processing procedure of arotation operation of a character object using the user terminal in FIG.1 .

FIG. 7 illustrates an example rotation operation of the characterobject.

FIG. 8 illustrates the example rotation operation of the characterobject.

FIG. 9 illustrates a rotation operation of a virtual object according toa comparative example.

DESCRIPTION OF EMBODIMENT

Hereinafter, embodiments of the present disclosure will be described indetail with reference to the accompanying drawings. In the descriptionof the drawings, the like or equivalent elements are denoted by the likereference numerals, and their explanations are not repeated.

[Configuration of User Terminal]

FIG. 1 illustrates an example hardware configuration of a user terminal10 that is a terminal device according to an embodiment. The userterminal 10 is a computer used by a user. In this embodiment, the userterminal 10 mainly has an image-capturing function of capturing a realspace, and a display function of displaying an augmented reality spaceimage representing an augmented reality space obtained by superimposinga virtual object on the captured real space.

The virtual object refers to a general object that does not actuallyexist in the real space, but gives the user an impression as if it werein the real space by being superimposed on the real space. Examples ofthe virtual object include a character object (i.e., an avatar)imitating an animate object, such as human or an animal, an objectimitating an inanimate object, such as a flower vase, and an objectaccording to a user operation in the augmented reality space (e.g., anobject imitating a cursor or any other pointer for pointing a positionin the augmented reality space).

As long as the user terminal 10 is a terminal device with both theimage-capturing function and the display function described above, theuser terminal 10 may be of any type with any configuration. For example,the user terminal 10 may be a high-function mobile phone (e.g., asmartphone), a tablet terminal, a wearable terminal (e.g., ahead-mounted display (HMD) or smart glasses).

As an example, the user terminal 10 includes a processor 101, a mainstorage unit 102, an auxiliary storage unit 103, a communication unit104, an image-capturing unit 105, a touch screen (display unit) 106, anda sensor 107 as hardware components.

The processor 101 is a computing device that executes an operatingsystem and application programs. The processor 101 may be a CPU or aGPU, for example, but the type of the processor 101 is not limited tothese.

The main storage unit 102 is a device configured to store thereinprograms for realizing the user terminal 10, and computation resultsoutput from the processor 101, or other data. The main storage unit 102is, for example, a ROM or a RAM.

The auxiliary storage unit 103 is typically a device capable of storinga larger amount of data than the main storage unit 102. The auxiliarystorage unit 103 is a non-volatile storage medium such as a hard disk ora flash memory, for example. The auxiliary storage unit 103 isconfigured to store various data and a virtual object operation programP1 for causing a computer to function as the user terminal 10. Forexample, the auxiliary storage unit 103 may store data related to thevirtual object.

The communication unit 104 is a device that establishes datacommunications with another computer via a communication network. Thecommunication unit 104 is, for example, a network card or a wirelesscommunication module. For example, the communication unit 104 maydownload data related to the virtual obj ect from an external serverdevice via a communication network such as the Internet, and store thedownloaded data in the auxiliary storage unit 103.

The image-capturing unit 105 is a device configured to capture an imageof a real space. The image-capturing unit 105 functions to captureimages of the real world. The image-capturing unit 105 is a camera, forexample. The image-capturing unit 105 may capture a moving image (video)or capture a still image (photograph). In a case of capturing a movingimage, the image-capturing unit 105 processes a video signal based on apredetermined frame rate so as to yield a time-sequential series offrame images as a moving image.

The touch screen 106 functions as an input interface that receives useroperations. For example, the touch screen 106 receives touch operations(e.g., tap operations, swipe operations, or multi-touch operations)performed by a user. The touch screen 106 also functions as an outputinterface that outputs data processed by the user terminal 10. The touchscreen 106 functions to display an augmented reality space imagerepresenting an augmented reality space including a real space capturedby the image-capturing unit 105 and a virtual object. That is, the userterminal 10 presents the augmented reality space to the user by causingthe touch screen 106 to display the augmented reality space image.

The sensor 107 is a sensor for detecting a direction and an amount ofmovement (displacement) of the user terminal 10. The sensor 107 is, forexample, an acceleration sensor. The type of the sensor 107 is howevernot limited thereto. For example, the sensor 107 may be a geomagneticsensor, a gyro sensor, or any other suitable sensor, or may beconfigured by a group of sensors including sensors of different types incombination.

The functional elements (e.g., a display control unit 11, adetermination unit 12, and an object control unit 13, which will bedescribed later) of the user terminals 10 are each implemented byretrieval of the virtual object operation program P1 and execution ofthe virtual object operation program P1 by the processor 101 or the mainstorage unit 102. The virtual object operation program P1 includes codesfor realizing the functional elements of the user terminal 10. Theprocessor 101 causes the communication unit 104, the image-capturingunit 105, the touch screen 106, or the sensor 107 to operate inaccordance with the virtual object operation program P1 to write andread data in and from the main storage unit 102 or the auxiliary storageunit 103. Through this processing, the functional elements of the userterminal 10 are realized.

The virtual object operation program P1 may be provided after beingfixedly recorded in a tangible recording medium, such as a CD-ROM, aDVD-ROM, or a semi-conductor memory. As an alternative, the virtualobject operation program P1 may be provided, as data signalssuperimposed on carrier waves, via a communication network.

[Functional Configuration of User Terminal]

FIG. 2 illustrates an example functional configuration of the userterminal 10. The user terminal 10 includes the display control unit 11,the determination unit 12, the object control unit 13, and a virtualobject information storage unit 14 as functional elements. The virtualobject information storage unit 14 stores therein virtual objectinformation related to the virtual object. The virtual objectinformation storage unit 14 may be the auxiliary storage unit 103described above, for example.

The virtual object information includes information for superimposingthe virtual object on the real space. For example, the virtual objectinformation includes arrangement information for specifying the positionand posture (i.e., angle) of the virtual object in the augmented realityspace. If the virtual object includes a plurality of parts (e.g., jointswhich will be described later, or the like) and position adjustment canbe performed individually for each of parts, the arrangement informationmay include information for specifying the position and posture of thepart. The information for specifying the position of each part of thevirtual object can be represented, for example, by coordinates in athree-dimensional coordinate space set in the real space captured andrecognized by the image-capturing unit 105.

The virtual object information may include information defining thespecifications of the virtual object. The specifications of a virtualobject refer to a rule or method of controlling the virtual object. Forexample, the specifications of a virtual object include at least one ofthe configuration (e.g., the shape and the size), the motion, and thesound of the virtual object. The data structure of the model data of thevirtual object is not limited and may be designed in any suitablemanner. For example, the model data on a character object may includeinformation on a plurality of joints and a plurality of bonesconstituting the character object, graphic data indicating an appearancedesign of the character object, and an ID which is an identifier foruniquely identifying the character object. Examples of the informationon joints and bones include three-dimensional coordinates of individualjoints and combinations of adjacent joints (i.e., bones). However, theconfiguration of the information is not limited to thereto and may bedesigned in any suitable manner.

The display control unit 11 causes the touch screen 106 to display anaugmented reality space image. For example, the display control unit 11continuously acquires the real space images representing a real space,which are captured by the image-capturing unit 105, and acquires (refersto) virtual object information stored in the virtual object informationstorage unit 14. The display control unit 11 superimposes a virtualobject on a real space image based on the arrangement informationincluded in the virtual object information. In this way, the augmentedreality space image is generated. The display control unit 11 causes thetouch screen 106 to display the augmented reality space image generatedin this manner.

The determination unit 12 determines at least a part of a virtual objectas an operation target. If the virtual object includes a plurality ofparts and operation can be performed individually for each of the parts,the part can be determined as the operation target. For example, thedetermination unit 12 may be configured to receive a user operation(e.g., tapping or holding-down) touching one of the parts of the virtualobject displayed on the touch screen 106 and determine the part touchedby the user as the operation target. Examples of the possible operationon the operation target include a moving operation and a rotationoperation which will be described later.

The object control unit 13 controls the operations of the operationtarget in the augmented reality space. The object control unit 13 canexecute a moving operation of moving the operation target in accordancewith a movement of the user terminal 10, and a rotation operation ofrotating the operation target in accordance with a user operation (e.g.,a swipe operation) on the touch screen 106. The moving operation orrotation operation of the virtual object is reflected on the augmentedreality space image on the touch screen 106 by the following mechanism,for example. For example, the object control unit 13 updates the virtualobject information related to the operation target stored in the virtualobject information storage unit 14 to a value representing the state(e.g., position or posture) of the operation target reflecting themoving operation or the rotation operation. After that, the displaycontrol unit 11 updates the state of the operation target on theaugmented reality space image on the basis of the virtual objectinformation updated. Accordingly, the augmented reality space imagereflecting the moving operation or the rotation operation is displayedon the touch screen 106.

[Moving Operation of Character Object]

An example operation of the user terminal 10 in a moving operation of acharacter object (i.e., a virtual object) in an augmented reality spacewill be described with reference to FIGS. 3 to 5 . FIG. 3 is a flowchartillustrating an example processing procedure of the moving operation ofthe character object using the user terminal 10. FIGS. 4 and 5illustrate an example moving operation of the character object. FIG. 4illustrates a state before moving the character object. FIG. 5illustrates a state after moving the character object.

In step S1, the display control unit 11 causes the touch screen 106 todisplay an augmented reality space image. More specifically, the displaycontrol unit 11 generates an augmented reality space image bysuperimposing the character object on the real space image, which hasbeen acquired by the image-capturing unit 105, based on the virtualobject information stored in the virtual object information storage unit14. The display control unit 11 then causes the touch screen 106 todisplay the augmented reality space image. In this way, as illustratedin FIG. 4 , an augmented reality space image 23 obtained bysuperimposing a character object 22 on a real space image 21 isdisplayed on the touch screen 106.

In step S2, the determination unit 12 determines at least a part of thecharacter object 22 as an operation target. In the example in FIG. 4 , aleft hand (e.g., a joint corresponding to the left hand) of thecharacter object 22 is determined as an operation target T by beingtapped by a user.

In step S3, the object control unit 13 detects the direction and amountof a movement of the user terminal 10 after the operation target T hasbeen determined. FIG. 5 illustrates an example where the user moves theuser terminal 10 leftward and closer to the user, as viewed from theuser. In FIG. 5 , v1 is a displacement vector of the user terminal 10(i.e., a vector indicating the direction and amount of the movement ofthe user terminal 10). The displacement vector v1 is, for example,detected by the sensor 107 and transmitted to the object control unit13. In this way, the object control unit 13 can detect the displacementvector v1.

In step S4, the object control unit 13 moves the operation target Tbased on the detected displacement vector v1. That is, the objectcontrol unit 13 adjusts the position of the operation target T based onthe displacement vector v1 in the augmented reality space (i.e., a spaceon the same scale as the real space). In FIG. 5 , v2 is a displacementvector of the operation target T (i.e., the vector indicating thedirection and amount of a movement of the operation target T). Theobject control unit 13 moves the operation target T so that thedisplacement vectors v1 and v2 are parallel to each other. That is, theobject control unit 13 moves the operation target T in the samedirection as the direction of the movement of the user terminal 10 inthe real space. This configuration allows the user to move the operationtarget T intuitively. That is, the movement of the user terminal 10 inthe real space allows a smooth movement of the operation target T notonly in the two-dimensional, vertical and horizontal directions but inthe three-dimensional directions including the depth.

In the example in FIG. 5 , the user moves the user terminal 10 leftwardand closer to the user, as viewed from the user so that the operationtarget T moves in the same direction as the direction (i.e., leftwardand closer to the user as viewed from the user) of the movement of theuser terminal 10. In this example, the bones connected to the operationtarget T (and other joints connected to the operation target T via thebones) also move in conjunction with the movement of the operationtarget T (i.e., the left hand of the character object 22). Thedirections and amounts of movements of the parts other than theoperation target T can be calculated out based on, for example, apositional relationship(s) between adjacent joints and an operation ruledetermined in advance based on rationality, consistency, and any othersuitable factors of a body motion. The positional relationship betweenof the joints and the operation rule are included in, for example, thevirtual object information related to the character object 22.

In step S4 described above, the object control unit 13 may be configuredto determine the amount of the movement of the operation target T basedon the detected amount of the movement of the user terminal 10 (i.e.,the magnitude |v1| of the displacement vector v1) and a parameter a setin advance. For example, the object control unit 13 may be configured todetermine, as the amount |v2| of the movement of the operation target T,a value obtained by increasing the detected amount |v1| of the movementof the user terminal 10 based on the parameter a. For example, theparameter a may be a coefficient, by which the amount |v1| of themovement of the user terminal 10 is multiplied, and which is larger thanone. In this case, the object control unit 13 determines the amount |v2|of the movement of the operation target T based on the followingequation (1).

|v2|=|v1|×a   (1)

For example, if the parameter a is five, the amount |v2| of the movementof the operation target T is 5 cm when the amount |v1| of the movementof the user terminal 10 is 1 cm. With such a configuration that theamount of a movement of the operation target T is amplified than theamount of an actual movement of the user terminal 10, the operabilityimproves. As described above, the amplification operation ofexaggerating the amount of a movement of the operation target T than theamount of an actual movement of the user terminal 10 is effective, forexample, for a case where it is desired that the position of theoperation target T be changed to relatively large extent. For example,in order to move the operation target T by 1 m, the moving operation ofthe operation target T can be made by the user simply moving the userterminal 10 at a shorter distance (e.g., 20 cm).

As an alternative, the object control unit 13 may be configured todetermine, as the amount |v2| of the movement of the operation target T,a value obtained by decreasing the detected amount |v1| of the movementof the user terminal 10 based on the parameter a. For example, theobject control unit 13 may be configured to diminish the amount of amovement of the operation target T than the amount of an actual movementof the user terminal 10, using the parameter a set to a value smallerthan 1 together with the value 1 and the equation (1). Such processingimproves the operability in fine position adjustment of the operationtarget T. That is, by intentionally reducing the sensitivity (i.e., theamount of the movement) of the operation target T according to themovement of the user terminal 10, accurate position adjustment of theoperation target T becomes possible.

Note that the parameter a is not limited to the coefficient used formultiplying the amount |v1| of the actual movement of the user terminal10 as in the equation (1) illustrated above. The parameter a may be afixed value determined in advance by the virtual object operationprogram P1 or may be a value variable by user settings or the like. Ifthe variable parameter a is adjustable, the user may adjust theparameter a in accordance with the situation of use. For example, asdescribed above, the parameter a can be set larger in order to move theoperation target T relatively largely, or smaller in order to adjustadjustment of the position of the operation target T finely.

Here, it is also conceivable that, in step S4, the operation target T isrotated in accordance with the rotation of the user terminal 10 byconfiguring such that the motion of the operation target T is fully inconjunction with the motion of the user terminal 10. However, therewould be such a drawback that, in case of employing the mechanism ofrotating the operation target T by causing the motion of the operationtarget T to be fully in conjunction with the motion of the user terminal10, the posture of the operation target T would reflect a minute change(rotation) in the orientation of the user terminal 10, even though sucha change is not intended by the user. In addition, in order to rotatethe operation target T from the front to the back as viewed from theuser by 180 degrees, this configuration requires the user to similarlyrotate the user terminal 10 by 180 degrees. In this case, theorientation of the user terminal 10 changes from the state with thetouch screen 106 facing toward the user to the state with the touchscreen 106 facing away from the user in the opposite direction. As aresult, the user cannot check the touch screen 106 after the rotationoperation (i.e., the user cannot promptly check the state of theoperation target T in the augmented reality space via the touch screen106).

To address the drawback, this embodiment is configured such that theobject control unit 13 moves the operation target T based on thedisplacement vector v1 without rotating the operation target T based onthe change in the attitude of the user terminal 10. That is, the objectcontrol unit 13 achieves the rotation operation of the operation targetT using a framework different from the moving operation of the operationtarget T (i.e., the operation based on the direction and distance of themovement of the user terminal 10). This can prevent the drawbackdescribed above.

[Rotation Operation of Character Object]

An example operation of the user terminal 10 performed for a rotationoperation of a character object (i.e., a virtual object) in an augmentedreality space will be described with reference to FIGS. 6 to 9 . FIG. 6is a flowchart illustrating an example processing procedure of therotation operation of the character object using the user terminal 10.FIGS. 7 and 8 illustrate an example rotation operation of the characterobject. FIG. 9 illustrates a rotation operation of a virtual objectaccording to a comparative example.

Steps S11 and S12 are the same as the processing (steps S1 and S2 inFIG. 3 ) in the moving operation of the character object 22. That is, instep S11, the display control unit 11 causes the touch screen 106 todisplay the augmented reality space image 23. Thereafter, in step S12,the determination unit 12 determines at least a part of the characterobject 22 as the operation target T. In the example in FIG. 7 , a righthand (e.g., a joint corresponding to the right hand) of the characterobject 22 is determined as an operation target T by being tapped by auser.

In step S13, the object control unit 13 receives rotation operationinformation. The rotation operation information is for instructing arotation operation of the operation target T. The rotation operationinformation includes information indicating the direction of rotation ofthe operation target T. The direction of rotation of the operationtarget T is specified based on an orthogonal coordinate system CS (seeFIG. 7 ), which is set with reference to the direction of capturing animage by the image-capturing unit 105. In other words, the orthogonalcoordinate system CS is set with reference to the point of view from theimage-capturing unit 105 (i.e., the angle of the augmented reality spaceimage 23 displayed on the touch screen 106). That is, the orthogonalcoordinate system CS changes in accordance with a change in the point ofview from the image-capturing unit 105.

In the example in FIG. 7 , the orthogonal coordinate system CS includesthree axes (X-, Y-, and Z-axes) orthogonal to each other. The X-(pitch)axis extends in the left-right direction with reference to the directionof image-capturing by the image-capturing unit 105 (i.e., the left-rightdirection of the augmented reality space image 23 displayed on the touchscreen 106). The Y-(roll) axis extends in the front-back direction withreference to the direction of image-capturing by the image-capturingunit 105 (i.e., the depth direction of the augmented reality space image23 displayed on the touch screen 106). The Z-(yaw) axis extends in theup-down direction with reference to the direction of image-capturing bythe image-capturing unit 105 (i.e., the up-down direction of theaugmented reality space image 23 displayed on the touch screen 106).

Upon receipt of the rotation operation information, the object controlunit 13 rotates the operation target T based on the rotation operationinformation (steps S14 to S17). In FIG. 7 , a state (A) illustrates astate before rotating the operation target T, while a state (B)illustrates a state after rotating the operation target T. FIG. 8illustrates an example rotation operation for changing the posture ofthe operation target T from the state (A) in FIG. 7 to the state (B) inFIG. 7 . In this example, in order to change the posture of theoperation target T from the state (A) in FIG. 7 to the state (B) in FIG.7 , the following rotation operations are executed: a rotation operation(hereinafter referred to as a “first rotation operation”), which is forchanging the posture of the operation target T from a state (A) in FIG.8 to a state (B) in FIG. 8 ; another rotation operation (hereinafterreferred to as “second rotation operation”), which is for changing theposture of the operation target T from the state (B) in FIG. 8 to astate (C) in FIG. 8 ; and yet another rotation operation (hereinafterreferred to as “third rotation operation”), which is for changing theposture of the operation target T from the state (C) in FIG. 8 to astate (D) in FIG. 8 .

The first rotation operation (i.e., from the state (A) to the state (B)in FIG. 8 ) can be executed by, for example, a horizontal swipeoperation on the touch screen 106. For example, if the object controlunit 13 detects a swipe operation of a user's finger on the touch screen106 sliding in the left-right direction (i.e., in the horizontaldirection) (swipe operation (horizontal) in step S14), the objectcontrol unit 13 rotates the operation target T about the Z-axis (stepS15). That is, the object control unit 13 receives the horizontal swipeoperation on the touch screen 106 as rotation operation informationindicating that rotation of the operation target T about the Z-axis isinstructed. As an example, the object control unit 13 receives arightward swipe operation as rotation operation information indicatingthat clockwise rotation of the operation target T with respect to thepositive direction of the Z-axis (i.e., upward) is instructed, and aleftward swipe operation as rotation operation information indicatingthat counterclockwise operation of the operation target T with respectto the positive direction of the Z-axis is instructed.

In the example in FIG. 8 , the object control unit 13 receives a swipeoperation to the right when the posture of the operation target T is inthe state (A), and changes the posture of the operation target T to thestate (B) based on the swipe operation. This changes the orientation ofthe fingertips of the right hand (i.e., the operation target T) of thecharacter object 22 bent inward substantially at a right angle from thestate (i.e., the state (A)) of pointing the left as viewed from theimage-capturing unit 105 to the state (i.e., the state (B)) of pointingthe front as viewed from the image-capturing unit 105.

The second rotation operation (i.e., from the state (B) to the state (C)in FIG. 8 ) can be executed by, for example, a vertical swipe operationon the touch screen 106. For example, upon detection of a swipeoperation of a user's finger on the touch screen 106 sliding in theup-down direction (i.e., in the vertical direction) (swipe operation(vertical) in step S14), the object control unit 13 rotates theoperation target T about the X-axis (step S16). More specifically, theobject control unit 13 receives the vertical swipe operation on thetouch screen 106 as rotation operation information indicating thatrotation of the operation target T about the X-axis is instructed. As anexample, the object control unit 13 receives an upward swipe operationas rotation operation information indicating that counterclockwiseoperation of the operation target T with respect to the positivedirection of the X-axis (i.e., to the right) is instructed, and adownward swipe operation as rotation operation information indicatingthat clockwise rotation of the operation target T with respect to thepositive direction of the X-axis is instructed.

In the example in FIG. 8 , when the posture of the operation target T isin the state (B), the object control unit 13 receives an upward swipeoperation and changes the posture of the operation target T to the state(C) based on the swipe operation. Accordingly, the right hand (i.e., theoperation target T) of the character object 22 bent inward substantiallyat a right-angle changes to stretch straight (i.e., the state (C)).

The third rotation operation (i.e., from the state (C) to the state (D)in FIG. 8 ) can be executed by, for example, a multi-touch rotationoperation on the touch screen 106. The multi-touch rotation operation isan operation of pressing two different points on the touch screen 106with two fingers (e.g., a thumb and an index finger) and sliding the twofingers in the same direction of rotation. For example, if the objectcontrol unit 13 detects the multi-touch rotation operation on the touchscreen 106 (multi-touch rotation operation in step S14), the objectcontrol unit 13 rotates the operation target T about the Y-axis (stepS17). More specifically, the object control unit 13 receives themulti-touch rotation operation on the touch screen 106 as rotationoperation information indicating that rotation of the operation target Tabout the Y-axis is instructed. For an example, the object control unit13 rotates the operation target T in the same rotation direction as thedirection of rotation in the multi-touch rotation operation. Morespecifically, the object control unit 13 receives a counterclockwisemulti-touch rotation operation on the touch screen 106 as rotationoperation information indicating that counterclockwise operation of theoperation target T with respect to the positive direction of the Y-axis(i.e., toward the back of the screen) is instructed, and a clockwisemulti-touch rotation operation on the touch screen 106 as rotationoperation information indicating that clockwise rotation of theoperation target T with respect to the positive direction of the Y-axisis instructed.

In the example in FIG. 8 , when the posture of the operation target T isin the state (C), the object control unit 13 receives a counterclockwisemulti-touch rotation operation on the touch screen 106 and changes theposture of the operation target T to the state (D) based on themulti-touch rotation operation. Accordingly, the right hand (i.e., theoperation target T) of the character object 22 stretching straight inthe vertical direction changes to slightly incline to the left (i.e.,the state (D)).

The amount of rotation (i.e., the angle) of the operation target T insteps S15 to S17 can be determined by, for example, the amount ofsliding (the distance of sliding) on the touch screen 106, the slidespeed, and any other suitable factor. In addition, the first and secondrotation operations described above may be performed simultaneously. Forexample, the user may perform a swipe operation of sliding a finger onthe touch screen 106 obliquely (e.g., toward the upper left, the lowerleft, the upper right, or the lower right). Such a swipe operation canbe decomposed into a horizontal swipe operation component and a verticalswipe operation component by vector decomposition. In this case, theobject control unit 13 may be configured to rotate the operation targetT about the Z-axis based on the horizontal swipe operation component andabout the X-axis based on the vertical swipe operation component.

[Advantages]

As described above, a user terminal 10 according to one aspect of thepresent disclosure is for presenting an augmented reality space to auser. The terminal device includes: an image-capturing unit 105configured to capture an image of a real space; a touch screen (displayunit) 106 configured to display an augmented reality space image 23representing the augmented reality space including the real spacecaptured by the image-capturing unit 105 and a virtual object; adetermination unit 12 configured to determine at least a part of thevirtual object as an operation target T; and an object control unit 13configured to control an operation of the operation target T in theaugmented reality space. The object control unit 13 detects a directionof a movement and an amount of a movement (a displacement vector v1 inthe embodiment described above) of the user terminal 10 after theoperation target T has been determined, and moves the operation target Tbased on the direction and the amount of the movement detected of theuser terminal 10.

A virtual object operation method according to an aspect of the presentdisclosure includes: displaying, on a touch screen 106, an augmentedreality space image 23 representing an augmented reality space includinga real space captured by an image-capturing unit 105 and a virtualobject; determining at least a part of the virtual object as anoperation target T; detecting a direction of an amount of a movement ofthe user terminal 10 after the operation target T has been determined;and moving the operation target T in the augmented reality space basedon the direction of movement detected and the amount of movementdetected of the user terminal 10.

A virtual object operation program P1 according to an aspect of thepresent disclosure is for causing a user terminal 10 to execute:displaying, on a touch screen 106, an augmented reality space image 23representing an augmented reality space including a real space capturedby an image-capturing unit 105 and a virtual object; determining atleast a part of the virtual object as an operation target T; detecting adirection and an amount of a movement of the user terminal 10 after theoperation target T has been determined; and moving the operation targetT in the augmented reality space based on the direction of movementdetected and the amount of movement detected of the user terminal 10.

In such aspects, the operation target T (at least a part of the virtualobject) is movable in the augmented reality space in accordance with thedirection and amount of the movement of the user terminal 10 in the realspace. That is, the aspects described above provide the user with amechanism capable of allowing intuitive position adjustment of theoperation target Tin the augmented reality space which is athree-dimensional space. More specifically, these aspects make itpossible for the user to intuitively adjust the position of theoperation target T in the augmented reality space by moving the userterminal 10 in a direction in which the user wants to move the operationtarget T.

A virtual object (e.g., the character object 22 in the embodimentdescribed above) displayed in the augmented reality space may include aplurality of parts, and the determination unit 12 may determine one ofthe parts as the operation target T. In this case, fine positionadjustment is possible for each of the parts of the virtual object.

The object control unit 13 may be configured to receive rotationoperation information including the direction of rotation of theoperation target T specified based on the orthogonal coordinate systemCS (see FIG. 7 ) including three axes (X-, Y-, and Z-axes), which is setwith reference to the direction of image-capturing by theimage-capturing unit 105 and used as the rotation axes of the operationtarget T. The object control unit 13 may then rotate the operationtarget T based on the rotation operation information. In this case,since the orthogonal coordinate system CS defining the three rotationaxes of the operation target T is determined with reference to thedirection of image-capturing by the image-capturing unit 105, anintuitive rotation operation of the operation target T is possible. Thatis, as illustrated in FIG. 8 , the user rotates the operation target Tin an intuitively determinable direction with reference to the directionin which the user views the augmented reality space via the touch screen106.

This will be described in detail with reference to the comparativeexample illustrated in FIG. 9 . In the comparative example, anorthogonal coordinate system CS1 defining three rotation axes (i.e., X-,Y-, and Z-axes) of a virtual object 30 is determined based not on thedirection of image-capturing by the image-capturing unit 105 (i.e., thedirection in which the user views the augmented reality space via thetouch screen 106) but on the posture of the virtual object 30. Here, asan example, the orthogonal coordinate system CS1 includes the X-axisperpendicular to a first surface 31 of a cubic virtual object 30, theY-axis perpendicular to a second surface 32, and the Z-axisperpendicular to a third surface 33. Here, the first surface 31, thesecond surface 32, and the third surface 33 are adjacent to each other.

FIG. 9 illustrates a state (B) after rotating the virtual object 30 in astate (A) in FIG. 9 about the X-axis. At this time, the rotation of thevirtual object 30 around the X-axis changes the orientations of the axes(Y-axis and Z-axis) other than the X-axis constituting the orthogonalcoordinate system CS1. FIG. 9 illustrates a state (C) after furtherrotating the virtual object 30 in the state (B) in FIG. 9 about theY-axis. At this time, the orientations of the axes (X-axis and Z-axis)other than the Y-axis constituting the orthogonal coordinate system CS1change. In this manner, in the orthogonal coordinate system CS1determined with reference to the posture of the virtual object 30, theorientations of the orthogonal coordinate system CS1 change inaccordance with a change in the posture of the virtual object 30 even ifthe direction of image-capturing by the image-capturing unit 105 doesnot change. Even in a same kind of rotation operation (e.g., anoperation of rotating the virtual object 30 about the Z-axis), thedirection of rotation of the virtual object 30 as viewed from the userchanges depending on the posture of the virtual object 30 at that time.For example, the direction of rotation viewed from the user at the timeof rotating the virtual object 30 about the Z-axis in the state (A) inFIG. 9 does not coincide with the direction of rotation viewed from theuser at the time of rotating the virtual object 30 about the Z-axis inthe state (B) or (C) in FIG. 9 . On the other hand, the orthogonalcoordinate system CS (see FIG. 8 ) used for the rotation operation ofthe operation target T in the embodiment described above is determinedwith reference to the direction of image-capturing by theimage-capturing unit 105 (i.e., the direction in which the user viewsthe augmented reality space via the touch screen 106) and does notchange depending on the shape of the operation target T. Therefore, theembodiment described above allows the user to always rotate theoperation target T with the same operation feeling regardless of theshape of the operation target T.

[Modifications]

The present disclosure has been described above in detail based on theembodiments. However, the present disclosure is not limited to theembodiments described above. The present disclosure may be modified invarious ways without departing from the spirit and scope thereof.

For example, some of the functions of the user terminal 10 describedabove may be executed by another computer device that is communicativewith the user terminal 10. That is, the user terminal 10 may causeanother computer device to execute some of the functions described aboveand receive the processing results from the other computer device,thereby implementing the functions described above.

The possible operations on the operation target T determined by thedetermination unit 12 are not limited to the moving operation and therotation operation described above. For example, the possible operationson the operation target T may include an operation of selecting theorientation of the operation target T from a plurality of shapesregistered in advance. For example, if a hand of the character object 22is the operation target T, the possible operations on the operationtarget T may include an operation of selecting a pose of the hand from aplurality of poses, such as “open,” “closed”, and a “V-sign”, registeredin advance.

While the embodiments described above are such that a part (section) ofthe character object 22 is determined as the operation target T, theoperation target T may be the entire character object 22. For example,for determining the location of the character object 22 in the augmentedreality space, or for like purposes, the entire character object 22 maybe determined as the operation target T. Further, a virtual object(e.g., an inanimate object or an object imitating an object imitating acursor or any other suitable pointer described above) other than thecharacter object 22 may be determined as the operation target T.

Two or more virtual objects may be arranged in the augmented realityspace. For example, assume that a virtual object (hereinafter simplyreferred to as an “apple”) imitating an apple and the character object22 are arranged in the augmented reality space. In such a case, theapple may be determined as the operation target T and a moving operationof putting the apple on the head of the character object 22 may beexecuted, for example. The moving operation by the object control unit13 described above allows the user to smoothly adjust the position ofthe apple by moving the user terminal 10 and thus performs the movingoperation described above intuitively and easily.

The part of the virtual object (e.g., character object 22) determined asthe operation target T is not limited to a hand illustratively describedin the embodiments described above. For example, a part, such as a foot,an elbow, a knee, or a waist (e.g., a part corresponding to a joint)other than a hand may be determined as the operation target T. Forexample, a part, such as eyes, a mouth, and a nose, other than a jointmay be determined as the operation target T.

Moreover, the augmented reality space image 23 may be such that a guidebe displayed to indicate a selectable part of the virtual object, whichis selectable as the operation target T (e.g., the guide may be a circlesurrounding the selectable part). Note that the guide may includeinformation indicating postures of the operation target T (e.g.,postures orienting in the frontward, sideward, upward, downwarddirections set in advance for the operation target T). In order toclarify the part selected as the operation target T, a guide forindicating the part selected as the operation target T may be displayedin a different display mode (e.g., a different color) from the guidesfor the parts not selected as the operation target T.

The processing procedure of the method (i.e., the virtual objectoperation method) executed by the user terminal 10 is not limited to theexamples in the embodiments described above (e.g., the flowcharts inFIGS. 3 and 6 ). For example, one or some of the above-described steps(processing) may be omitted, or the steps may be executed in anotherorder different from the one in the example. Any two or more of theabove-described steps may be combined, or part of the steps may bemodified or deleted. As an alternative, other steps may be performed inaddition to the steps described above.

DESCRIPTION OF REFERENCE CHARACTERS

10 User Terminal (Terminal Device), 11 Display Control Unit, 12Determination Unit, 13 Object Control Unit, 14 Virtual ObjectInformation Storage Unit, 21 Real Space Image, 22 Character Object(Virtual Object), 23 Augmented Reality Space Image, 101 Processor, 105Image-Capturing Unit, 106 Touch Screen (Display Unit), 107 Sensor, CSOrthogonal Coordinate System, T Operation Target

1-9. (canceled)
 10. A terminal device configured to present an augmented reality space to a user, comprising: an image-capturing unit configured to capture an image of a real space; a display unit configured to display an augmented reality space image representing the augmented reality space, the augmented reality space including the real space captured by the image-capturing unit and a virtual object; a determination unit configured to determine at least a part of the virtual object as an operation target; and an object control unit configured to control an operation of the operation target in the augmented reality space, wherein the object control unit is configured to: detect a direction and an amount of a movement of the terminal device after the operation target is determined; and move the operation target based on the detected direction and amount of the movement of the terminal device.
 11. The terminal device of claim 10, wherein the object control unit is configured to determine an amount of a movement of the operation target based on the detected amount of the movement of the terminal device and a parameter set in advance.
 12. The terminal device of claim 11, wherein the object control unit determines, as the amount of the movement of the operation target, a value obtained by increasing the detected amount of the movement of the terminal device based on the parameter.
 13. The terminal device of claim 11, wherein the object control unit determines, as the amount of the movement of the operation target, a value obtained by decreasing the detected amount of the movement of the terminal device based on the parameter.
 14. The terminal device of claim 10, wherein the object control unit moves the operation target based on the detected direction and amount of the movement of the terminal device without rotating the operation target in accordance with a change in an attitude of the terminal device.
 15. The terminal device of claim 10, wherein the virtual object includes a plurality of parts, and the determination unit determines, as the operation target, one of the plurality of parts of the virtual object.
 16. The terminal device of claim 10, wherein the object control unit is configured to: receive rotation operation information including a direction of rotation of the operation target, the direction of rotation specified based on an orthogonal coordinate system including three axes, the three axes set with reference to a direction of image-capturing by the image-capturing unit and used as rotation axes of the operation target; and rotate the operation target based on the rotation operation information.
 17. A virtual object operation method, comprising: displaying, on a display unit included in a terminal device, an augmented reality space image representing an augmented reality space, the augmented reality space including a real space captured by an image-capturing unit included in the terminal device and a virtual object; determining at least a part of the virtual object as an operation target; detecting a direction and an amount of a movement of the terminal device after the operation target is determined; and moving the operation target in the augmented reality space based on the detected direction and amount of the movement of the terminal device.
 18. The method of claim 17, further comprising determining an amount of a movement of the operation target based on the detected amount of the movement of the terminal device and a parameter set in advance.
 19. The method of claim 18, wherein determining the amount of the movement of the operation target includes determining a value obtained by increasing the detected amount of the movement of the terminal device based on the parameter.
 20. The method of claim 18, wherein determining the amount of the movement of the operation target includes determining a value obtained by decreasing the detected amount of the movement of the terminal device based on the parameter.
 21. The method of claim 17, wherein moving the operation target includes moving the operation target based on the detected direction and amount of the movement of the terminal device without rotating the operation target in accordance with a change in an attitude of the terminal device.
 22. The method of claim 17, wherein the virtual object includes a plurality of parts, and the method further comprises determining, as the operation target, one of the plurality of parts of the virtual object.
 23. The method of claim 17, further comprises: receiving rotation operation information including a direction of rotation of the operation target, the direction of rotation specified based on an orthogonal coordinate system including three axes, the three axes set with reference to a direction of image-capturing by the image-capturing unit and used as rotation axes of the operation target; and rotating the operation target based on the rotation operation information.
 24. A non-transitory computer-readable medium storing a virtual object operation program that, when executed, causes a terminal device to execute: displaying, on a display unit included in a terminal device, an augmented reality space image representing an augmented reality space, the augmented reality space including a real space captured by an image-capturing unit included in the terminal device and a virtual object; determining at least a part of the virtual object as an operation target; detecting a direction and an amount of a movement of the terminal device after the operation target is determined; and moving the operation target in the augmented reality space based on the detected direction and amount of the movement of the terminal device.
 25. The non-transitory computer-readable medium of claim 24, wherein the program causes the terminal device to further execute determining an amount of a movement of the operation target based on the detected amount of the movement of the terminal device and a parameter set in advance.
 26. The non-transitory computer-readable medium of claim 25, wherein determining the amount of the movement of the operation target includes determining a value obtained by increasing or decreasing the detected amount of the movement of the terminal device based on the parameter.
 27. The non-transitory computer-readable medium of claim 25, wherein moving the operation target includes moving the operation target based on the detected direction and amount of the movement of the terminal device without rotating the operation target in accordance with a change in an attitude of the terminal device.
 28. The non-transitory computer-readable medium of claim 24, wherein the virtual object includes a plurality of parts, and the program causes the terminal device to further execute determining, as the operation target, one of the plurality of parts of the virtual object.
 29. The non-transitory computer-readable medium of claim 24, wherein the program causes the terminal device to further execute: receiving rotation operation information including a direction of rotation of the operation target, the direction of rotation specified based on an orthogonal coordinate system including three axes, the three axes set with reference to a direction of image-capturing by the image-capturing unit and used as rotation axes of the operation target; and rotating the operation target based on the rotation operation information. 